What is the effect of increasing the number of search threads/parallelism on a single node with limited CPU cores?

Increasing search threads on a single node has the classic parallelism curve: initially throughput scales roughly linearly as more cores are used, then it plateaus as the cores are saturated, and then it degrades as oversubscription causes context switching, cache thrashing, and memory-bandwidth contention. For a CPU-bound search workload, the sweet spot is at or slightly below the number of physical cores, and going beyond that is rarely beneficial. The specific shape of the curve depends on whether the workload is compute-bound or memory-bound. Distance computations are compute-heavy, so more threads help until the cores are busy. But the HNSW traversal involves random memory access, which is memory-bandwidth-bound, so beyond a certain point additional threads contend for the same memory bus and the marginal throughput is near zero or negative. On on-disk collections, the bottleneck may be I/O rather than CPU, in which case more threads can increase I/O queue depth and help - up to the point where the disk's queue is saturated.

There is an interaction with segments and shards that is easy to miss. A single search over a collection is parallelized across segments and across shards, so the amount of work available to parallelize is bounded by the number of segments and the number of shards on the node. If you set the thread count higher than the available parallelism, the extra threads have nothing to do and you get no benefit - or you get coordination overhead. This is why the right thread count depends on the collection layout, not just on the CPU count. On a node with a few large segments, there is less parallelism to exploit than on a node with many small segments. Similarly, a collection with 4 shards on a node can use at most 4-way parallelism for a single query at the shard level, regardless of how many cores are available.

The trade-off is throughput against latency and fairness. More threads can reduce the latency of a single query if there is work to parallelize, but they can also increase the latency of other queries by stealing cores. If the node is also running the optimizer or handling writes, the search threads compete with those workloads, and oversubscribing makes the contention worse. My rule is to size the search thread pool close to the physical core count, leave headroom for the optimizer and for the OS, and measure throughput and p99 under representative load rather than under a single-query benchmark. The common mistake is benchmarking a single query with high thread count and concluding that high thread count is good, without measuring the system under concurrent load. The second mistake is assuming that adding threads will reduce latency for a single query when the collection has few segments or shards - there is simply not enough parallel work. The third mistake is ignoring the optimizer: on a node where the optimizer is also running, the effective core count for search is lower than the total, and oversubscribing amplifies the contention. Version note: how Qdrant exposes and uses search thread configuration has changed across releases; in some versions it is a server-level setting, in others it is per-request, and the default behavior differs.

Version-dependent: the configuration surface for search threads (server-level vs per-request, the default value, and whether the optimizer shares the same pool) has changed across Qdrant releases. If you are tuning thread counts, check the actual settings exposed by your version and measure under your own load pattern rather than relying on a fixed recommendation.